A storage is computer-readable media capable of storing data in blocks. Storages face a myriad of threats to the data they store and to their smooth and continuous operation. In order to mitigate these threats, a backup of the data in a storage may be created to represent the state of the source storage at a particular point in time and to enable the restoration of the data at some future time. Such a restoration may become desirable, for example, if the storage experiences corruption of its stored data, if the storage becomes unavailable, or if a user wishes to create a second identical storage.
A storage is typically logically divided into a finite number of fixed-length blocks. A storage also typically includes a file system which tracks the locations of the blocks that are allocated to each file that is stored in the storage. The file system also tracks the blocks that are not allocated to any file. The file system generally tracks allocated and unallocated blocks using specialized data structures, referred to as file system metadata. File system metadata is also stored in designated blocks in the storage.
Various techniques exist for backing up a source storage. One common technique involves backing up individual files stored in the source storage on a per-file basis. This technique is often referred to as file backup. File backup uses the file system of the source storage as a starting point and performs a backup by writing the files to a destination storage. Using this approach, individual files are backed up if they have been modified since the previous backup. File backup may be useful for finding and restoring a few lost or corrupted files. However, file backup may also include significant overhead in the form of bandwidth and logical overhead because file backup requires the tracking and storing of information about where each file exists within the file system of the source storage and the destination storage.
Another common technique for backing up a source storage ignores the locations of individual files stored in the source storage and instead simply backs up all allocated blocks stored in the source storage. This technique is often referred to as image backup because the backup generally contains or represents an image, or copy, of the entire allocated contents of the source storage. Using this approach, individual allocated blocks are backed up if they have been modified since the previous backup. Because image backup backs up all allocated blocks of the source storage, image backup backs up both the blocks that make up the files stored in the source storage as well as the blocks that make up the file system metadata. Also, because image backup backs up all allocated blocks rather than individual files, this approach does not necessarily need to be aware of the file system metadata or the files stored in the source storage, beyond utilizing minimal knowledge of the file system metadata in order to only back up allocated blocks since unallocated blocks are not generally backed up.
An image backup can be relatively fast compared to file backup because reliance on the file system is minimized. An image backup can also be relatively fast compared to a file backup because seeking is reduced. In particular, during an image backup, blocks are generally read sequentially with relatively limited seeking. In contrast, during a file backup, blocks that make up individual files may be scattered, resulting in relatively extensive seeking.
Multiple source storages can be backed up to the same destination storage. When these source storages are similar, many of the source storages may contain redundant files, blocks, or data between the various systems. For example, if multiple source storages utilize the same commercial operating system, such as WINDOWS® 8.1, they may store a common set of system files which will have identical blocks. If these source storages are backed up to the same destination storage, these identical blocks will be stored in the destination storage multiple times, resulting in redundant blocks. Redundancy in a destination storage may increase the overall size requirements of destination storage and increase the bandwidth overhead of transporting blocks to the destination storage. To reduce the redundancy in the storage and free extra blocks, a deduplication system can be used with a hash table to determine which blocks are currently stored.
One common problem with deduplication systems and their hash tables is the requirement to make a trade-off between monetary price and ingestion performance based on the structure and media used for the hash table. Typically, either fast data ingestion rates can be obtained at a high monetary cost using fast media or price can be reduced by using a different media with worse ingestion performance. In the typical use case, the hash table is written sporadically across many blocks instead of being written in batches to the storage device. While the highest data rates can be achieved using physical random-access memory (RAM), RAM is relatively expensive. If a flash storage device is used to store the hash table, the flash storage device will suffer performance and reliability penalties as the same regions of the flash storage device are rewritten many times, which happens frequently when hashes are relatively evenly spread across the flash storage device. In addition, while reading from a flash storage device is a relatively quick procedure, writing to a flash storage device is relatively slow and wears on the flash storage device.
The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one example technology area where some embodiments described herein may be practiced.